Information bottleneck

Claude Shannon‘s Rate Distortion theory formalizes the trade off between compression and the preservation of meaning (e.g., how higher-level categories can compress and preserve). Namely, given data $X$ and compressed representation $\hat{X}$, it seeks encodings that minimize $$R + \lambda D = I(X; \hat{X}) + \lambda \mathbb{E}[d(X, \hat{X})],$$ where $R$ is the rate (number of bits) and $D$ is distortion.

• [ ] why do we want to minimize Mutual information here?

Information bottleneck theory then argues to minimize the following when compressing $X$ into $Z$ while preserving information about relevant variable $Y$: $$\min I(X; Z) - \beta I(Z; Y)$$